Big data in the acoustic phonetic analysis

Lecture for class: Data science in the Humanities

This tutorial will demonstrate how to analyse audio data for acoustic phonetic studies in R. It is mainly intended to demonstrated possible workflows. The topics covered are:

• phonetic databases, the case of emuR and EMU-SDMS, the EMU Speech Database Management System
• Sample data: the LOD database
• Queries and requeries
• Inscpect the database: serve()
• Calculate duration for vowel categories
• Vowels formants and visualisations with ggplot2
• Vowel explorer
• Calculating the Pillai distance

Preamble

The generated HTML page of this tutorial is available here.

This tutorial is organised as an R Markdown notebook. To execute the code within this notebook (filename Lecture.Rmd) it has to be opend in RStudio. When executing code, the results appear beneath the code. In order to do so, you need to have R and RStudio installed. When this R notebook is loaded into RStudio, you can excecute chunks by clicking the Run button within the chunk or by placing your cursor inside it and pressing Cmd+Shift+Enter, allowing you to experiment with the code. This handout contains all output of the code (tables, visualisations etc.). The easiest way to work with this R code is to clone the entire project from this GitHub repository.

The following libraries will be needed and have to be installed.

library(tidyverse)
library(ggiraph)
library(cowplot)
library(emuR)
library(tools)
library(rio)
library(ggplot2)
library(magrittr)
library(ggiraph)
library(htmltools)
library(shiny)
library(joeyr)
library(knitr)

1 The LOD database

The database contains the audio recordings from the Lëtzebuerger Online Dictionnaire (available here, spoken by one female speaker. The audio files have been automatically segmented with the MAUS tools. We thus have a database conisting of textual data, basically words, and the corresponding audio data. The audio data is segmented into words and phonetic segments (sounds).

This database has been created beforehand. Infos how to create such a database is explained in the EMU-SDMS manual.

While we will be working with the full database (26.000 recordings), a small demo database lod_emuDB has been created with 500 recordings which can be used for individual testing.

We start with loading the database and give an overview of structure and content.

# load database
db = load_emuDB("/Users/peter.gilles/Documents/_Daten/LOD-emuDB/lod_emuDB")

INFO: Checking if cache needs update for 1 sessions and 26093 bundles ...
INFO: Performing precheck and calculating checksums (== MD5 sums) for _annot.json files ...
INFO: Nothing to update!

#db = load_emuDB("lod_emuDB")

# display the overview of the structure and content
summary(db)

Name:    lod 
UUID:    c87793c0-012a-11e9-874b-68b599b5deb4 
Directory:   /Users/peter.gilles/Documents/_Daten/LOD-emuDB/lod_emuDB 
Session count: 1 
Bundle count: 26093 
Annotation item count:  515638 
Label count:  639542 
Link count:  489545 

Database configuration:

SSFF track definitions:

Level definitions:

Link definitions:

Tracks in emuR are acoustic representation of the speech signal, here dft for the waveform (time-amplitude representation) and praatFms for the formant measures of vowels (see below).

Levels in an eumR database stand for level of interlinked linguistic information. bundle is the entire audio file, ORT stands for the orthographical representation of the audio file segmented in its single words. MAU is the segmentation of all phonetic segment (=sounds) of all ORT segments in bundle.

The hierarchical structure of these levels is expressed in the link definitions as ONE-TO-MANY.

2 Database queries

An emuR database can be queried with a powerful query engine. The first example is a simple query for one word, Aarbecht.

sl = query(db, query = "[ ORT == 'Aarbecht']")
sl

The result is a segment list (sl), containing various information about the found item (time, level, name, database info). The result of the query can also be displayed in the EMU Speech Database Management System.

serve(db, seglist = sl)

The GUI will open in the Viewer pane of RStudio or you can open it in a browser (Chrome preferred).

Here we can also display the hiearchical structure for this database item, which is accessed during queries. bundle is the top-level, representing the entire audio file.

The ORT level contains the nodes for the individual words in the bundle, here the two words Aarbecht and Aarbechten. The dependend level then is MAU (=Munich Automatic Unit) representing the single sounds of the words in ORT.

Two aspects render emuR query system extremly powerful: the use of regular expressions (including negation and other extensions) and the combinated query on different levels of the database.

Let’s try more complex queries:

• regular expression, operator =~, words beginning with Aarbecht...

sl = query(db, query = "[ ORT =~ 'Aarbecht.*']")
sl

Select the vowel [aː] in all words beginning with Aarbecht… Note that in the segment list the label now has changed to the vowel and the respective start-end information is now only for this sound [aː].

sl = query(db, query = "[ ORT =~ 'Aarbecht.*' ^ #MAU=='aː']")
sl

NA

• query a sequence of sounds, e.g. e followed by k (Méck), by using the sequence operator ->.

sl = query(db, query = "[ MAU == e -> MAU == k ]")
# print only the first 100 rows
sl[1:100, ]

NA

• in the sequence e->k, query only the vowel. Use the result modifier #.

sl = query(db, query = "[ #MAU == e -> MAU == k ]")

NA

# query all phoneme items that occur
# at the start of a syllable
sl <- query(db, query = "[End(ORT, MAU) == TRUE & MAU == p ]")
sl

# retrieve all word that contain five segments
sl <- query(db, "[Num(ORT, MAU) == 5]")
sl

• using requery the results from a previous query can be further specified

########################
# requery_seq()

# query all "n" phonetic items
sl_n = query(db, "MAU == m")

# sequential requery (left shift result by 1 (== offset of -1))
# and hence retrieve all phonetic items directly preceeding
# all "m" phonetic items
sl_req_n <- requery_seq(db, 
            seglist = sl_n, 
            offset = -1)
sl_req_n

• groups auf sounds can be grouped together to label groups, e.g. all long monophthongs ("iː", "uː", "aː", "oː", "ɔː", "ɛː", "eː") or all short monophthongs ("i", "u", "ɑ", "o", "æ", "e", "ə", "ɐ").

# query all "m" phonetic items
sl = query(db, "MAU == longMonophthongs")
sl

With the query the user can compile the data frame from the database which then forms the subset for the phonetic analysis. We can select e.g. all instances of certain (or all) vowels, specifying the context before or after etc. etc.

Of course, querying for individual segments in the audio file like words or sounds is possible only, if this information has been added to the database before.

3 Signal processing in R

The first task is to extract the time-amplitude waveform representation (oscillogram) for certain single sounds, e.g. some long aː.

# query all "aː" phonetic items
# 
sl = query(db, "MAU == aː")
# instead of all 7,000 vowels take only the first 6
sl = sl[1:6, ]

# get "fm" track data for these segments
# (verbose = F is only set to avoid additional output)
a_vowels = get_trackdata(emuDBhandle = db,
                         seglist = sl,
                         ssffTrackName = "fm",
                         verbose = FALSE)

Error in get_trackdata(emuDBhandle = db, seglist = sl, ssffTrackName = "fm",  : 
  The emuDB object lod does not have any ssffTrackDefinitions called fm

The data for the oscillogram is stored in the SSF track dft, which is extracted by get_trackdata. The result is another R dataframe.

# get "dft" track data for these segments
a_vowels = get_trackdata(emuDBhandle = db,
                         seglist = sl,
                         ssffTrackName = "MEDIAFILE_SAMPLES",
                         verbose = TRUE)

The emusegs/emuRsegs object passed in refers to bundles with in-homogeneous sampling rates in their audio files! Here is a list of all refered to bundles incl. their sampling rate: 

  INFO: parsing 6 wav segments/events

  |                                                                        
  |                                                                  |   0%
  |                                                                        
  |===========                                                       |  17%
  |                                                                        
  |======================                                            |  33%
  |                                                                        
  |=================================                                 |  50%
  |                                                                        
  |============================================                      |  67%
  |                                                                        
  |=======================================================           |  83%
  |                                                                        
  |==================================================================| 100%

The oscillograms for these 6 vowel instances can then be visualised with ggplot2.

# plot oscillogram
ggplot(data = a_vowels) + 
  aes(y = T1, x = times_rel) + 
  geom_line() + 
  facet_wrap(~sl_rowIdx + labels)

# query all words beginning with 'Fluch'
# 
sl = query(db, query = "[ ORT =~ 'Fluch.*']")

# get "f0" track data for these segments, in this case calculated on the fly
fluch_words = get_trackdata(emuDBhandle = db,
                         seglist = sl,
                         # using emuR's Michel Scheffers’ Modified Harmonic
                         # Sieve algorithm 
                         onTheFlyFunctionName = "mhsF0",
                         verbose = TRUE)

The emusegs/emuRsegs object passed in refers to bundles with in-homogeneous sampling rates in their audio files! Here is a list of all refered to bundles incl. their sampling rate: 

  INFO: applying mhsF0 to 12 segments/events

  |                                                                        
  |                                                                  |   0%
  |                                                                        
  |======                                                            |   8%
  |                                                                        
  |===========                                                       |  17%
  |                                                                        
  |================                                                  |  25%
  |                                                                        
  |======================                                            |  33%
  |                                                                        
  |============================                                      |  42%
  |                                                                        
  |=================================                                 |  50%
  |                                                                        
  |======================================                            |  58%
  |                                                                        
  |============================================                      |  67%
  |                                                                        
  |==================================================                |  75%
  |                                                                        
  |=======================================================           |  83%
  |                                                                        
  |============================================================      |  92%
  |                                                                        
  |==================================================================| 100%

In the corresponding graphs then the track for the fundamental frequency (f0) for some isolated words will be drawn.

# plot f0 tracks
ggplot(data = fluch_words) + 
  # define the df columns for the x and y values
  aes(y = T1, x = times_rel) + 
  # line chart
  geom_line() + 
  # sl_rowIdx groups all rows in a dataframe belonging to the same segment
  # labels contains the label of the segment
  facet_wrap(~ sl_rowIdx + labels)

If needed, you can check the content of your segment list with the EMU WebApp running serve(seglist = sl).

4 Calculate duration for vowel categories

The study of duration of speech segments is a standard task in acoustic phonetics. Let’s see how to solve this in R. Thanks to the label groups which are available in the database, we have easy access to e.g. all shortMonophthongs and all longMonophthongs.

# query all "m" phonetic items
longMonophthongs = query(db, "MAU == longMonophthongs")
shortMonophthongs = query(db, "MAU == shortMonophthongs")

# print the count
length(longMonophthongs)

[1] 16

length(shortMonophthongs)

[1] 16

The data frame contains per segment a start and end column, which allows for simple calculation of the segment duration.

# calculate durations
duration_long = longMonophthongs$end - longMonophthongs$start
duration_short = shortMonophthongs$end - shortMonophthongs$start

# calculate mean and standard deviation
paste("The mean duration for long monophthongs in the databse is", round(mean(duration_long)), "ms. The standard deviation is:", sd(duration_long), ".")

[1] "The mean duration for long monophthongs in the databse is 153 ms. The standard deviation is: 62.9196191781937 ."

paste("The mean duration for short monophthongs in the databse is", round(mean(duration_short)), "ms. The standard deviation is:", sd(duration_short), ".")

[1] "The mean duration for short monophthongs in the databse is 106 ms. The standard deviation is: 51.4225215439996 ."

Instead of having these broad categories long and short we can break them down to the individual vowels. The dataframes just created contain the individual vowels already in the labels column and can thus be easily utilised.

head(longMonophthongs)

First, we create a aggregated datafrom with all vowel segments. This aggregated dataframe now contains 127,065 rows (= long/short vowels).

# add new columns for length, either long or short
longMonophthongs$length <- "long"
shortMonophthongs$length <- "short"

# then merge the two data frame into a single one
all_vowels <- rbind(longMonophthongs, shortMonophthongs)

# add a new colum for segment duration
all_vowels$duration <- round(all_vowels$end - all_vowels$start)

Using the powerful library dplyr, we can now calculate the means for all vowels (= labels).

duration_vowels <- all_vowels %>%
  group_by(labels, length) %>%
  summarise(mean = mean(duration), sd = sd(duration))

duration_vowels

Finally, visualise the means in a bar chart

ggplot(data = duration_vowels, aes(x= labels, y= mean, fill = length)) +
  geom_bar(stat="identity", position=position_dodge()) +
  geom_errorbar(aes(ymin=mean-sd, ymax=mean+sd), width=.2,
                 position=position_dodge(.9)) +
  theme_minimal()

From here on the duration values can be utilised in further statistical analyses.

5 Vowels formants and visualisations with ggplot2

Formant frequencies are the acoustic representation of the vowel’s quality. Of these, the first two formants are crucial for the vowels identity, i.e. F1 and F2. The values for the formants are already calculated in the database and will now be extracted for all the vowels our dataframe all_vowels.

As it takes long to do this, the dataframe has been precompiled and saved to file …

saveRDS(td_vowels, file ="td_vowels.RDS")

… to be simply loaded.

td_vowels <- readRDS("td_vowels.RDS")

Formants are calculated for every sampling point of the audio file. Thus, the dataframe td_vowels now contains some 2,3 million rows/values.

For illustration purposes, the formants for the long vowels in three example words are selected into a dataframe (maachen, Viz, Kuuscht).

formant_examples <- td_vowels %>%
  filter(sl_rowIdx == "722" | sl_rowIdx == "21702" | sl_rowIdx == "11191")

These three vowels then are visualised as line charts.

ggplot(data=formant_examples) +
  geom_smooth(aes(x = times_norm, y = T1, col = labels, group = labels)) +
  geom_smooth(aes(x = times_norm, y = T2, col = labels, group = labels)) +
  labs(x = "Duration (normalized)", y = "F1 + F2 (Hz)") +
  ggtitle("Example formants F1, F2 the three corner vowels") +
  facet_wrap(~ labels)

vowel_midpoints = td_vowels %>% 
  filter(times_norm == 0.4)

6 Calculate the Pillai distance

6.1 from https://joeystanley.com/blog/a-tutorial-in-calculating-vowel-overlap

7 Vowel explorer

Try it here.

knitr::include_app("https://petergill.shinyapps.io/shinyplay/")